
US006615182B1 



(12) United States Patent (io) Patent No.: US 6,615,182 Bl 

Powers et al. (45) Date of Patent: Sep. 2, 2003 



(54) SYSTEM AND METHOD FOR DEFINING 
THE ORGANIZATIONAL STRUCTURE OF 
AN ENTERPRISE IN A PERFORMANCE 
EVALUATION SYSTEM 

(75) Inventors: Michael C Powers, Irving, TX (US); 

Douglas A. Sudberry, Grapevine, TX 
(US); James A. Eiler, Irving, TX (US); 
Robert S. Bennett, Franklin, TN (US); 
Clifford R. Phillips, Bedford, TX (US) 

(73) Assignee: e-talk Corporation, Irving, TX (US) 

( * ) Notice: Subject to any disclaimer, the term of this 
patent is extended or adjusted under 35 
U.S.C. 154(b) by 0 days. 

(21) Appl. No.: 09/110,106 

(22) Filed: Jul. 1, 1998 

Related U.S. Application Data 

(60) Provisional application No. 60/084,794, filed on May 8, 
1998. 

(51) Int. CI. 7 G06F 17/60 

(52) U.S. CI 705/7; 705/8; 705/9; 705/11; 

709/1; 709/2; 709/9; 709/10 

(58) Field of Search 705/1, 7, 9, 8, 

705/11, 12; 709/1, 2, 9, 10 

(56) References Cited 

U.S. PATENT DOCUMENTS 

4,476,535 A 10/1984 Loshing et al 364/480 

4,764,867 A * 8/1988 Hess 395/356 

5,017,917 A * 5/1991 Fisher et al 340/14.1 

5,172,313 A * 12/1992 Schumacher 705/7 

5,233,513 A * 8/1993 Doyle 705/7 

5,241,621 A 8/1993 Smart 395/51 

5,321,621 A 6/1994 Sainen 364/470 

5,349,663 A * 9/1994 Bailey 707/100 

5,406,477 A * 4/1995 Harhen 395/500.27 

5,493,489 A * 2/1996 Tamaki et al 705/8 

5,500,795 A 3/1996 Powers et al 364/401 

5,551,880 A 9/1996 Bonnstetter et al. 



EP 
EP 
GB 
WO 



5,617,342 A * 4/1997 Elazouni 395/500.27 

5,630,069 A * 5/1997 Flores et al 705/7 

5,655,118 A * 8/1997 Heindel et al 707/103 

5,675,745 A * 10/1997 Oku et al 705/7 

5,684,964 A 11/1997 Powers et al 395/211 

5,726,914 A 3/1998 Janovski et al. 

5,734,837 A * 3/1998 Flores et al 705/7 

5,784,452 A 7/1998 Carney 

5,799,286 A * 8/1998 Morgan et al 705/30 

5,809,131 A * 9/1998 Ichikawa et al 705/7 

5,812,135 A * 9/1998 Kotchey 345/356 

5,909,669 A 6/1999 Havens 

6,088,679 A * 7/2000 Barkley 705/8 

6,298,349 Bl * 10/2001 Toyoshima ct al 345/853 

FOREIGN PATENT DOCUMENTS 

0 587 290 A2 7/1993 
0 644 510 A2 9/1994 
2221068 A 1/1990 
WO 97/31320 8/1997 

OTHER PUBLICATIONS 



Search Report re: PCT/ US 99/ 10143, Sep. 16, 1999. . 

Teknekron Infoswitch Corporation, "P&Q Review!™ Pro- 
ductivity and Quality Performance Evaluation, Getting 
Started Guide " Version 2 #151-0801-002, Jun. 1995. 

(List continued on next page.) 

Primary Examiner — V. Millin 

Assistant Examiner— Jagdish N Patel 

(74) Attorney, Agent, or Firm— Baker Botts L.L.P. 



(57) 



ABSTRACT 



An organizational structure of an enterprise is defined in a 
performance evaluation system by storing a plurality of 
user-defined levels. A user-defined hierarchy is stored for the 
levels. A plurality of user-defined members are also stored. 
Each member is associated with a level. The performance 
evaluation system is operable to generate the organizational 
structure of the enterprise based on the user-defined levels, 
hierarchy, and members. 

17 Claims, 9 Drawing Sheets 
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SYSTEM AND METHOD FOR DEFINING user-defined levels, hierarchy and members and the assign- 

THE ORGANIZATIONAL STRUCTURE OF ment of members to levels. 

AN ENTERPRISE IN A PERFORMANCE More specifically, in accordance with one embodiment of 

EVALUATION SYSTEM tne present invention, the user-defined levels are stored by 

S listing the levels in a first database table. In this embodiment, 

RELATED APPLICATIONS the user-defined hierarchy is stored in a second database 

table. The hierarchy includes an assignment for each lower 

This application claims the benefit of U.S. Provisional i eve i [ n tne hierarchy to a higher level in the hierarchy. The 

Application Serial No. 60/084,794, entitled "METHOD user-defined members are stored by listing the members in 

AND SYSTEM FOR DETERMINING THE PERFOR- i(j a third database table. The members are each associated with 

MANCE OF A CALL CENTER AGENT", filed May 8, a level by storing a user-assigned level for the members in 

1998, which is hereby incorporated by reference. the third database table. 

This application is related to copending U.S. patent appli- Technical advantages of the present invention include 

cation Ser. No. 09/110,108 entitled "SYSTEM AND providing an improved performance evaluation system. In 

METHOD FOR GENERATING AN EVALUATION IN A 15 particular, the performance evaluation system uses a web- 

PERFORMANCE EVALUATION SYSTEM"; copending base architecture that allows performance evaluation meth- 

U.S. patent application Ser. No 09/110,109 entitled "SYS- odolo Sy 10 be standardized throughout an enterprises In 

TEM AND METHOD FOR PROVIDING ACCESS PRIV1- addition, users can take advantage of the Internet or other 

LEGES FOR USERS IN A PERFORMANCE EVALUA- network to remotely access the performance evaluation 

TION SYSTEM"; copending U.S. patent application Ser. M s y stcm and complete member evaluations. Members can 

No. 09/110,103, entitled "SYSTEM AND METHOD FOR ^° remotely track their progress. 

IMPORTING PERFORMANCE DATA INTO A PERFOR- Another technical advantage of the present invention 

MANCE EVALUATION SYSTEM"; and copending U.S. includes providing a performance evaluation system that can 

patent application Ser. No. 09/110,107 entitled "SYSTEM be customized to fit the organizational structure of an 

AND METHOD FOR GENERATING RESULTS IN A 1$ enterprise. In particular, users may define levels and a 

PERFORMANCE EVALUATION SYSTEM". " hierarchy for the levels within the performance evaluation 

system that match that of the enterprise. As a result, quality 

TECHNICAL FIELD OF THE INVENTION and productivity tools are customized to fit the structure of 

^ . . ,i , the enterprise. 

This invention relates generally to evaluation tools, and ...... .„ l j i 

. i i , , j , r , a . 4L „ Other technical advantages will be readily apparent to one 

more particularly to a system and method lor denning the 30 5» . j • a 

: A . , . r . ■• _r skilled in the art from the following figures, description, and 

organizational structure ot an enterprise in a performance . . ° ° r 



evaluation system. 

BACKGROUND OF THE INVENTION 



claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 
For a more complete understanding of the present inven- 



Evaluation tools are used to gather and record productiv- 35 tion and its advantages, reference is now made to the 

ity and quality statistics for a business or other organization. following description taken in conjunction with the accom- 

Analysis of such information allows inefficiencies and other panying drawings in which: 

problems of the business to be identified and corrected. In FIG. 1 is a block diagram illustrating a performance 

this way, business performance is maximized. evaluation system in accordance with one embodiment of 

Traditionally, evaluation tools are implemented on stand- tne present invention; 

alone systems that are unable to effectively communicate FIG. 2 is a block diagram illustrating an example call 

with other resources in a network environment. Other prob- center using the performance evaluation system of FIG. 1; 

lems include the use of predefined organizational structures FIG. 3 illustrates details of the organization tables of FIG. 

that cannot be modified to match the organizational structure 45 1 in accordance with one embodiment of the present inven- 

of a business. In addition, privileges, evaluations, and tion; 

reports are generally hard -coded for each evaluation tool. FIG. 4 illustrates details of the privilege tables of FIG. 1 

Such customization leads to high implementation and j n accordance with one embodiment of the present inven- 

administration cost. tion; 

SUMMARY OF THE INVENTION 50 ™|- 5 illUS " aleS det f S / P ] a " ^ °f FIG " * " 

accordance with one embodiment or the present invention; 

In accordance with the present invention, a performance FIG. 6 illustrates details of the productivity tables of FIG. 

evaluation system is provided that substantially eliminates 1 in accordance with one embodiment of the present inven- 

or reduces disadvantages or problems associated with the tion; 

previously developed systems. In particular, the present 55 FIG. 7 illustrates details of the data import tables of FIG. 

invention provides a performance evaluation system in 1 in accordance with one embodiment of the present inven- 

which the organizational structure of an enterprise is accu- tion; 

rately defined to custom fit quality and productivity tools to FIG. 8 illustrates details of an evaluation file stored in the 

the structure of the enterprise. evaluation tables of FIG. 1 in accordance with one embodi- 

In one embodiment of the present invention, the organi- 60 ment of the present invention; and 

zational structure of an enterprise is defined in a perfor- F IG. 9 illustrates details of the filter tables of FIG. 1 in 

mance evaluation system by storing a plurality of user- accordance with one embodiment of the present invention, 
defined levels. A user-defined hierarchy is stored for the 

levels. A plurality of user-defined members are also stored. DETAILED DESCRIPTION OF THE 

Each member is associated with a level to define the 65 INVhNUON 

organizational structure of the enterprise. The performance FIG. 1 illustrates a performance evaluation system 10 in 

evaluation system may generate a user view based on the accordance with one embodiment of the present invention. 
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As described in more detail below, the performance evalu- 
ation system 10 uses productivity and quality data to evalu- 
ate the performance of an individual, group, process or other 
suitable type of item or operation. 

Referring to FIG. 1, the performance evaluation system 5 
10 is deployed on a three -tiered architecture. The three - 
tiered architecture includes client space 12, server applica- 
tion space 14 and database space 16. The client space 12 is 
implemented on a client platform 18 such as a work station, 
personal computer or other device capable of communicat- 1° 
ing between a user and a server. The server application and 
database spaces 14 and 16 are implemented on a server 
platform 20 such as a personal computer or other device 
capable of being remotely accessed over a network. 

The client and server platforms 18 and 20 are connected 15 
by a network 22. The network 22 may comprise the Internet 
or other wide area network (WAN), an enterprise intranet or 
other a local area network (LAN), or other suitable type of 
link capable of communicating data between the client and 
server platforms 18 and 20. 20 

The client space 12 includes a web-browser 30 having a 
graphical user interface (GUI) 32. The graphical user inter- 
face 32 displays web pages 34 downloaded over the network 
22 from the server application space 14. The web pages 34 25 
may use HTML or Active X controls 36. The Active X 
controls 34 provide a rich interface 32 with superior script- 
ing control. 

The server application space 14 includes a server engine 
40, business objects 42, reports 44, charts 46, and a data 30 
importer 48. The server engine 40 includes active server 
pages (ASP) 50 that include server-side components and 
scripting. The server-side components are specialized Active 
X components executed prior to delivery of web pages to the 
client space 12. The server-side scripting accesses the inter- 35 
faces of the server-side components. Use of the active server 
pages 50 allows the web pages to be updated in response to 
changing system and database settings. 

The business objects 42 call stored procedures in the 
database space 16 and perform preprocessing and expres- 40 
sions processing functions. In a particular embodiment, the 
business objects 42 include an organization object, a plan 
object, an evaluation object, an import service object, and a 
results object. In this embodiment, the organization object 
provides an interface for basic organizational functionality. 45 
The organization object also provides an interface to a class 
of services table giving the client space 12 access to allowed 
functions. The plan object provides an interface for the 
manipulation and use of evaluation guidelines. The evalu- 
ation object provides interfaces for performing evaluations. 50 
The import service object schedules and retrieves external 
data and places it into a delimited file. The import service 
object also provides a basic interface to the configuration 
settings. The results object receives information from the 
client space 12 for selecting reports based on the requested 55 
filters. Each of the objects may comprise one or more 
objects. 

The reports 44 may include both detail and summary 
reports. The detail reports provide information for members 
and elements of each member. In a particular embodiment, 60 
the detail reports include a productivity and quality report, 
a quality evaluation report, a productivity analysis report, a 
quality question report, a productivity element report, and a 
detail evaluation report. The productivity and quality report 
presents actual scores, maximum scores, and percentages of 65 
maximum for both quality and productivity by an evaluation 
date for each member with group averages for all hierarchi- 
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cal levels. The quality evaluation report presents notes, 
actual scores, maximum score, and percentages of maximum 
for each question of an evaluation with subtotals by perfor- 
mance areas, evaluation, member, and any additional hier- 
archical levels. The productivity analysis report presents 
actual scores, maximum score, and percentage of maximum 
for each equation of productivity analysis with subtotals by 
member and by any additional hierarchical levels. The 
quality and question report presents actual scores for each 
question along with the total actual score, maximum score, 
and percentage of maximum for each performance area with 
subtotals by evaluation, member, and any additional hierar- 
chical levels. The productivity element report presents 
actual data along with the equation-produced actual score, 
maximum score, and percentage of maximum for each 
element of productivity data with subtotals by member and 
any additional hierarchical levels. The detail evaluation 
report presents evaluation date, time, and status of all 
evaluations for a member subtotal by status, date, member, 
and evaluator. 

The summary reports provide average scores for a par- 
ticular member or level. In a particular embodiment, the 
summary reports include a productivity and quality report, a 
quality evaluation report, a productivity analysis report, a 
quality question report and a productivity and quality com- 
parison report. The productivity and quality report presents 
average scores for both quality and productivity for each 
member with group averages for all hierarchical levels. The 
quality evaluation report presents average scores by perfor- 
mance areas, evaluation, member, and any additional hier- 
archical levels. The productivity analysis report presents 
average scores for each equation of productivity data with 
subtotals by member and any additional hierarchical levels. 
The quality question report presents the total score for each 
question and the percentage of maximum scores for each 
performance area. The productivity and quality comparison 
report presents average scores, maximum scores and per- 
centage of maximum for both quality and productivity by 
member with comparison averages for a chosen hierarchical 
level. Additional detail and summary reports can be created 
by selecting various categories of information. 

The charts 46 display graphic and textual information 
useful in determining the overall performance of a member 
and identifying ways to improve that performance. In a 
particular embodiment, the charts 46 include sets of pro- 
ductivity and quality charts, productivity charts and quality 
charts. The productivity and quality charts plot productivity 
and quality scores for date, member, users and hierarchical 
levels. The productivity charts present productivity data 
plotted against time for members, users, and other hierar- 
chical levels. The quality charts present quality data plotted 
against time for members, users, questions and hierarchical 
levels. 

The data importer 48 imports productivity data from 
external sources such as a telephony switch. The data 
importer 48 includes preprocessors 52 and a work area 54. 
The preprocessors 52 are tools used to create an instruction 
file for interpreting the information in a data file. The 
instruction files are operable to parse data files. In one 
embodiment, the preprocessors 52 include Nortel 
Datastream, Rockwell Spectrum IMS, and Monarch. The 
Nortel Datastream preprocessor is used for converting 
Nortel Datastream report information into a delimited for- 
mat. The Rockwell Spectrum IMS preprocessor is used for 
converting Rockwell report information. The Monarch pre- 
processor can be used to create custom preprocessors. 
Accordingly, the performance evaluation system 10 can be 
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configured to import any type of suitable data file 56 from Referring to FIG. 2, the call center 100 includes a server 

an external device 58. 102 connected to a plurality of clients 104 through a local 

The preprocessors 52 create a delimited file 60 from a data area network (LAN) 106. The call center 100 includes a 

file 56. In the delimited file 60, columns of data are each system administrator (user 1), a service manager (user 10), 

separated by a delimiter character such as a comma, hyphen, s a technical manager (user 20), a sales manager (user 30), a 

colon, and the like. The delimiter acts as reference point product A manager (user 31), and a product B manager (user 

telling the data importer 48 where one column of data stops 35). Service agents (members 11 and 12) report to the 

and a new column starts. If the data file 56 is already in a scrvicc managcr . Technical agents (members 21 and 22) 

delimited format, a preprocessor 52 is not used. report to me technical manager. Product A agents (members 

The delimited file 60 is a temporary file stored in the work 10 32 an d 33) report to the product A manager. Product B agents 

area 54. As described in more detail below, the delimited file ( raem bers 36 and 37) report to the product B manager. In 

60 is moved to the database space 16 and the productivity tfae ^ A and B managers re ort t0 ^ sales 

data mapped into the database based on the configuration manaeer 

defined in the database. Accordingly, the evaluation process « T Jr*.„ c . . ,10^^. 

is streamlined by having data in a central location for FIG. 3 illustrates details of the organization tables 82. The 

analysis 15 organization tables 82 allow an enterprise to set up the 

Hie database space 16 includes a database manager 70 Performance evaluation system 10 to fit the organizational 

and a database 72. The database manager 70 calls stored slruclure of the enterprise. Thus the organization tables 82 

procedures 80 to access the database 72. Hie stored proce- ma / s "f u re number of user-defined levels in any user- 

dures 80 are a precompiled collection of SQL statements and , n defined ^ this way, quality and productivity tools 

optional control-of-flow statements stored under a name and 2 ° arc CU5lomized for the cn terpnse. 

processed as a unit. The stored procedures 80 are stored Referring to FIG. 3, the organization tables 82 include a 

within a database 72, can be executed with one call from the level table 120, an interlevel assignment table 122, member 

database manager 70, and allow user-declared variables, tables 124, a user table 126, a user-member table 128, a 

conditional executions, and other programming features. keyword table 130, and a keyword assignment table 132. 

In addition to the stored procedures 80, the database 72 ™ e leveI tabie 120 iists aI1 ^ levcls of ^ enterprise by a 

includes organization tables 82, privileges tables 84, plans * evel ID. The levels are a collection of members who can be 

tables 86, data import tables 88, productivity tables 90, S rou P ed to g ether - ™ e level IDs may be any suitable iden- 

evaluation tables 92 and filter tables 94. As described in tifier operable to uniquely identify the levels. For the call 

more detail below, the organization tables 82 allow an 30 center 100 of FIG. 2, the levels are the company, the service 

enterprise to set up the performance evaluation system 10 to S rou P* technical group, the sales group, the product A 

correspond to its organizational structure. The privileges team > and product B team. 

tables 84 store user access privileges based on views of the The interlevel assignment table 122 assigns a lower level 
organizational structure and on class of services. The plan in the enterprise hierarchy to a higher level in the hierarchy, 
tables 86 store questions, performance areas, and guidelines 35 In the interlevel assignment table 122, a first field identifies 
for generating performance evaluations in the performance a higher level to which a lower level is assigned and a second 
evaluation system 10. The data import tables 88 store field identifies the lower level. The labels "first" and "sec- 
configurations for importing productivity data into the per- ond" are used to distinguish between fields and not to 
formance evaluation system 10. The productivity tables 90 specify a particular order for the fields. The lower levels may 
store productivity data for members of the performance 40 be assigned to a next higher level in the hierarchy or to each 
evaluation system 10. The evaluation tables 92 store higher level in the hierarchy. The levels are identified in the 
responses and scores for completed evaluations. The filter interlevel assignment table 122 by the level IDs. For the call 
tables 94 store filters for sorting data and displaying results center 100 of FIG. 2, the service, technical, and sales group 
in the reports 44 and charts 46. levels are assigned to the company level and the product A 

In a particular embodiment, the client and server plat- 45 and B team levels are assigned to the sales group level, 

forms 18 and 20 are each a personal computer connected via The member tables 124 list in a first field all the members 

the Internet. In this embodiment, the client and server 18 and of the enterprise by a member ID. The members are those 

20 may each operate using MICROSOFT WINDOWS NT individuals of the enterprise on which functions of the 

version 4.0 that provides TCP/IP connectivity over the system 10 are performed. Each member ID may be unique 

Internet. The web-browser 30 may be Internet Explorer 3.2x 50 codes assigned by the performance evaluation system 10 

or higher. The server engine 40 may be Internet Information when the member is first defined in the system 10. 

Server 3.0 or higher. The database may be a relational In the member tables 124, a second field assigns the 

database and the database manager 70 an SQL Server 6.5. member to a level. Additional fields provide member 

The web-base architecture of the performance evaluation attributes, such as first and last name, password, comments, 

system 10 allows performance evaluation methodology to be 5s and the like. In another embodiment, some or all of the 

standardized throughout an enterprise. In addition, users can member attributes may be stored in a separate table. For the 

take advantage of the Internet or other network to remotely call center 100 of FIG. 2, service agents (members 11 and 

access the performance evaluation system and complete 12) are assigned to the service group, technical agents 

member evaluations. Members can also remotely track their (members 21 and 22) are assigned to the technical group, 

progress. 60 product A agents (members 31 and 32) are assigned to the 

FIG. 2 illustrates an exemplary call center 100 using the product A team, and so on through product B agent (member 

performance evaluation system 10. Although the perfor- 37) assigned to product B team. 

mance evaluation system 10 will be described in connection The user table 126 lists all the users of the enterprise by 

with the exemplary call center 100, it will be understood that a user ID. The users are people in the performance evalu- 

the performance evaluation system 10 may be used for other 65 ation system 10 that perform the evaluations and that carry 

types of enterprises, including other types of companies, out the various tasks associated with the evaluation process 

businesses, groups, and organizations. such as defining all of the information needed to perform the 
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evaluations. The user IDs may be any suitable identifier 
operable to uniquely identify the users. For the call center 
100 of FIG. 2, the users are the system administrator (user 
1), the service manager (user 10), the technical manager 
(user 20), the sales manager (user 30), the product A 
manager (user 31), and the product B manager (user 35). 

The users-member table 128 cross-references user IDs 
with member IDs for users who are also members in the 
system 10. A user is also a member whea the user reports to 
and is evaluated by a user higher in the enterprise hierarchy. 
For the call center 100 of FIG. 2, users 31 and 35 report to 
user 30. Accordingly, users 31 and 35 are cross-referenced 
as members in the users-member table 128. 

The keyword table 130 lists all the keywords defined in 
the performance evaluation system 10 by a keyword ID. The 
keywords provide an easy way to classify members and 
other items of the performance evaluation system 10. The 
keywords can also be used to filter performance data for 
analysis or printing reports. The keyword IDs may be any 
suitable identifier operable to uniquely identify the key- 
words in the system 10. For the call center 100 of FIG. 2, the 
keywords may relate to the experience of an agent or to an 
agency from which the agent was hired. In this way, the 
performance of the agents may be compared based on their 
experience, the agency from which they were hired, and the 
like. 

The keyword assignment table 132 assigns keywords to a 
member. In the keyword assignment table 132, a first field 
identifies a member by a member ID and a second field 
identifies a keyword assigned to the member by a keyword 
ID. One or more keywords may be assigned to each of the 
members. For the call center 100 of FIG. 2, if the service 
agent (member 11) is hired from the AAAgency and has two 
years* experience, keywords "Temp-AA Agency" and "2 
years experience" are assigned to the service agent (member 

ii). 

FIG. 4 illustrates details of the privilege tables 84. The 
privilege tables 84 assign each user a view and a class of 
services. The view specifies the levels and members of an 
organizational structure to which the user is allowed access. 
The class of services specifies services of the performance 
evaluation system 10 that the user is allowed to perform. The 
user has access privileges to perform services within the 
user's class of services for levels and members within the 
user's view. In this way, access privileges may be easily 
updated and maintained for the user in response to changes 
in allowed services or organizational structure. As a result, 
system administration cost is reduced. 

Referring to FIG. 4, the privilege tables 84 include a user 
view table 140, a class of service assignment table 142, and 
a class of service template table 144. The user view table 140 
specifies levels and members of the organizational structure 
to which the user is allowed access. 

In the user view table 140, a first field identifies a user by 
user ID, a second field identifies a level or member associ- 
ated with the user by the level or member ID, a third field 
identifies whether the item in the second field is a level or a 
member, and a fourth field enables the stored association 
between a user and a level or member in the user's view. 
Thus, in this embodiment, each user is associated with each 
level in the user's branch of the organizational structure and 
with members that report to the user. Each association 
between a user and a level or member may be enabled or 
disabled in the user's view and deselected using column 196 
to be excluded from the user's view. In another embodiment, 
the user view table 140 may include only associations that 
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are included in the user's view. In this embodiment, the 
fourth field is omitted. 

For the call center 100 of FIG. 2, the system administrator 
(user 1) is associated with each level and member in the 

5 performance evaluation system 10. All of the associations 
for the system administrator are enabled and thus included 
in the system administrator's view to allow the system 
administrator to access each level and member of the 
system. The service manager (user 10) is associated with 

10 each level in the service manager's branch of the organiza- 
tional structure and with members that report to the service 
manager. Thus, the service manager is associated with the 
company and service group levels and with the service 
agents (members 10 and 11) that report to the service 

35 manager. The associations between the service manager, the 
service level and the service agents are enabled to allow the 
service manager to access the level and members to config- 
ure and perform performance evaluations and other func- 
tions for the members and level. The association with the 

20 company level is not enabled and thus excluded from the 
service manager's view. Similarly, the product B manager 
(user 35) is associated with the company, sales group, and 
product A and B team levels and with the product B agents 
(members 36 and 37) that report to the product B manager. 

25 The associations with the product B team level and product 
B agents are enabled and included in the product B man- 
ager's view. Access to the company and the sales group 
levels are not enabled and thus excluded from the product B 
manager's view. 

30 The user view table 140 may be manually populated by 
the system administrator or other user having suitable access 
privileges or may be generated based on the organizational 
structure defined by the organization tables 82, In the latter 
embodiment, a user assignment table is used to assign each 

35 user to a level. Based on that assignment, the organization 
tables 82 are interrogated to determine the levels within the 
user's branch of the organizational structure and the mem- 
bers that report to the user. This information is then used to 
populate the user view table 140. After the table is 

40 populated, associations are to be included in the user's view. 
The class of service assignment table 142 assigns a class 
of service template to a user. In the class of service assign- 
ment table 142, a first field identifies the user by the user ID 

45 and a second field identifies the class of service template by 
a service template ID. The service template ID may be any 
suitable identifier operable to uniquely identify the service 
templates. Use of the service templates allows a particular 
class of services to be defined and reused for any number of 

50 users. For example, for the call center 100 of FIG. 2, group 
and team templates may be defined and associated with the 
group and team managers, respectively. Accordingly, the 
performance evaluation system 10 is more efficiently con- 
figured and administration cost is reduced. 

55 The class of service template table 144 specifies the class 
of services for each service template. The class of service 
template table 144 includes a first field identifying a service 
template by the service template ID. Additional fields are 
included to specify the services to be included in the service 

60 template. The class of service template table 144 also 
includes a field for identifying a specific user by the user ID. 
This is used to generate a custom class of services for the 
user rather than a service template. 
The services are predefined for the performance evalua- 

65 tion system 10. In a particular embodiment, the services may 
include administer productivity values, administer produc- 
tivity data elements, access charting and reporting and 
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administer filters, administer keywords, administer member operable to uniquely identify the performance areas. For the 

information, administer class of service, administer evalu* call center 100 of FIG. 2, the performance areas may include 

ation expressions, configure data import settings, import on-call greeting and call closing. 

data, administer organizational structure, perform reviews, The question assignment table 156 assigns questions to 

add or modify reviews, delete reviews, administer perfor- s the performance areas. In the question assignment table 156, 

mance areas and questions, browse my reviews, browse my a nrst field identifies a performance area by a performance 

performed reviews, browse all reviews, and administer user arca ID and a second field identifies a question assigned to 

information. me performance area by a question ID. One or more 

- .„' . i t *i c ( l i * ui t>u i questions may be assigned to each of the performance areas. 

FIG. 5 illustrates details of the plan tables 86. The plan ^ ' uestions & may be reused bet ^ een performance 

tables 86 store questions, performance areas constructed of io ^ For e ^ performance areas 2, 3 and 4 may each 

a set of questions, and guidelines constructed of a set of include question 3 

performance areas The questions performance areas, and ^ expression ' tabIe 158 lists m a first field all the 

guidelines are used to generate performance evaluations for expressions in ±c performance evaluation system 10 by an 

members of the performance evaluation system 10. The use expression ID. The expression IDs may be any suitable 

and sharing of questions by multiple performance areas and ^ identifier operable to uniquely identify the expressions. A 

the use and sharing of performance areas by multiple second field specifies a mathematical equation for the 

guidelines allows evaluations to be efficiently defined and expression. The equation may contain data elements, 

automatically generated. In addition, members in a group are numeric values, mathematical operations or mathematical 

evaluated against the same set of criteria to provide fairness functions. In a particular embodiment, the mathematical 

and objectivity. As a result, managers can perform 20 operations include addition, subtraction, multiplication and 

consistent, objective evaluations efficiently. division. The mathematical functions include sum, average, 

Referring to FIG. 5, the plan tables include a question minimum, maximum, and peak, 

table 150, an answer types table 152, a performance area The expression assignment table 160 assigns expressions 

table 154, a question assignment table 156, an expression to the performance areas. When associated with a perfor- 

table 158, an expression assignment table 160, a guideline 25 mance area, the expressions are used to compute a produc- 

table 162, and a performance area assignment table 164. The tivity score for that performance area. In the expression 

question table 150 defines questions that will be used in the assignment table 160, a first field identifies a performance 

performance evaluations. The question table 150 includes a area by a performance area ID and a second field identifies 

first field identifying the question by a question ID, a second an expression assigned to the performance area by an 

field providing a title for the question, a third field specifying expression ID. One or more expressions may be assigned to 

text for the question, and a fourth field assigning a weight to each of the performance areas. In addition, an expression 

the question. The question ID may be any suitable identifier may be reused between performance areas. For example, 

operable to uniquely identify the questions. performance areas 1 and 6 may each include expression 1. 

For the call center 100 of FIG. 2, a first question may be 35 The guideline table 162 lists all the guidelines in the 

directed toward schedule adherence and ask "How often performance evaluation system 10 by a guideline ID. Each 

does the agent work on the originally assigned shift?" and a guideline combines one or more performance areas that are 

second question may be directed toward promptness and ask of interest for a particular evaluation. The guideline IDs may 

"How often is the agent late for work?". Relative weights are be any suitable identifier operable to uniquely identify the 

assigned to the question depending on their importance to 4Q guidelines. For the call center 100 of FIG. 2, the perfor- 

the enterprise. Thus, if promptness is more important to the mance areas may include technical group, product A team, 

enterprise than schedule adherence, the second question will product B team, and the like. 

be assigned a greater weight. The performance area assignment table 164 assigns per- 

The answer types table 152 associates each question with formance areas to the guidelines. In the performance area 

a predefined answer type and attributes for that answer type. 45 assignment table 164, a first field identifies a guideline by a 

In the answer types table 152, a first field identifies the guidelines ID and a second field identifies a performance 

question by the. question ID, a second field identifies the area assigned to the guidelines by a performance area ID. 

answer type by an answer type ID, and a third field identifies One or more performance areas may be assigned to each of 

the attributes for the answer type. The answer type IDs may the guidelines. In addition, performance areas may be reused 

be any suitable identifier operable to uniquely identify the 50 between guidelines. For the call center 100 of FIG. 2, for 

answer types. example, the service, technical, product A team, and product 

In a particular embodiment, defined answer types include B team guidelines may each include performance area 1. 

yes/no, drop-down lists, horizontal scroll bar, edit box, and FIG. 6 illustrates details of the productivity tables 90. The 

radio button answers. The yes/no answer type asks for either productivity tables 90 store productivity data for members of 

a yes or a no response. The drop-down list asks for one 55 the performance evaluation system 10. The productivity data 

answer from a list of possible answers. The horizontal scroll is used by the expressions in the performance areas to 

bar answer type asks for an answer on a continuum for an calculate productivity scores. 

overall rating. The edit box answer type asks for a numeric Referring to FIG. 6, the productivity tables 90 include a 

answer. The radio buttons ask for one answer from a list of data element table 166 and a productivity table 168, The data 

possible answers. Attributes for the answer types include 60 element table 166 lists all the data elements in the perfor- 

where appropriate, target values, labels, maximum and mini- mance evaluation system 10 by a data element ID. The data 

mum values, and the like. elements are items of information used to measure the 

The performance area table 154 lists all the performance productivity of members in the performance evaluation 

areas in the performance evaluation system 10 by a perfor- system 10. The data element IDs may be any suitable 

mance area ID. Each performance area is a grouping of 65 identifier operable to uniquely identify the data elements, 

questions that relates to a particular area of job performance. The productivity table 168 includes a first field identifying 

The performance area IDs may be any suitable identifier members with a member ID, a second field providing dates 
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for the records, a third field identifying the data element for 
the record by a data element ID, and a fourth field storing the 
value of the data element. 

FIG. 7 illustrates details of the data import tables 88. The 
data import tables 88 import productivity data into the 
performance evaluation system 10 from external devices and 
automatically insert it into the production table. As a result, 
the evaluation process is streamlined by having data in a 
centralized location for data analysis. 

Referring to FIG. 7, the data import tables 88 include 
configuration tables 170, an import template assignment 
table 172, and an import template table 174. Each configu- 
ration table 170 provides information for identifying and 
mapping data from the delimited file 60 to the productivity 
table 90. 

The configuration table 170 includes a first field identi- 
fying a column in the delimited file 60 containing a data item 
corresponding to a data element, a second field identifying 
a data name, a third field identifying a data type, a fourth 
field identifying a data format, and a fifth field mapping the 
data item to the data element in the productivity table 90. In 
a particular embodiment, the data types include identity, 
date, duration and numeric types. An identity column pro- 
vides identifying information such as a name or code. The 
date column provides the date of the information. The 
duration column shows an amount of time for the informa- 
tion. A numeric column contains a value. 

The import template assignment table 172 assigns an 
import template to a configuration. In the import template 
assignment table 172, a first field identifies the configuration 
by a configuration ID and a second field identifies the import 
template by an import template ID. The configuration and 
import template IDs may be any suitable identifiers operable 
to uniquely identify the configurations and import templates, 
respectively. Use of the import templates allows a process 
for imported data to be defined and reused for any number 
of configurations. Accordingly, the performance evaluation 
system 10 is more efficiently configured and administration 
cost is reduced. 

The import template table 174 specifies preprocessor and 
other information for importing the data file 56. The import 
template table 174 includes a first field identifying the 
import templates by the import template IDs. A second field 
identifies a preprocessor, if any, to be applied to the data file 
56 to generate the delimited file 60. A third field identifies 
the delimiter used in the delimited file 60. 

In operation, productivity data is imported by selecting a 
configuration, selecting users to determine how the data will 
be applied (members assigned to selected users receive the 
productivity data), specifying how the date will be 
determined, selecting the data file 56 to import by selecting 
a drive and the file name, specifying how to match produc- 
tivity data to members (by name or code), and selecting 
whether or not to replace existing productivity information. 
In response to the import request, the data importer 48 
retrieves the requested data file 56 and if the import template 
associated with the configuration requests a preprocessor 52, 
applies the preprocessor 52. Using the delimiter information 
in the import template for the configuration and the mapping 
information in the configuration table, the data importer 48 
inserts the productivity data into the productivity tables 90 
for members in the user's view. 

FIG. 8 illustrates details of an evaluation stored in the 
evaluation tables 92. As previously described, the evaluation 
is automatically generated based on guidelines, performance 
areas, and questions selected for the evaluation. 
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Referring to FIG. 8, the evaluation includes a member ID 
180 for whom the evaluation was completed, a date 182 on 
which the evaluation was completed, and a plurality of 
performance areas 184. The performance areas 184 each 

s include one or more questions 186 and their associated 
answer types 188. As the evaluation is completed, responses 
190 are stored for each question and used to calculate a 
quality score 192 for the performance area 184. The quality 
score 192 is a measure of "goodness" for the member's 

10 performance. 

The quality score 192 for a performance area 184 is 
calculated based on the relative weight of the questions 186 
in the performance area 184. The score of responses 190 to 
all questions 186 in the performance area 184 are summed 

15 and divided by the sum of the relative weights. This pro- 
duces a quality score 192 for the performance area 184. 
Mathematically, this is expressed in the following way: 

v^ (( achievedi - worst; \ \ 

20 > I * relWeigty 

\\ target; -worst; ) ) 

Z relWeighTi 

25 where 

n is the number of applicable questions in the perfor- 
mance area 

i ranges over the answered questions in the performance 

30 

achieved is the number of points the member actually 

earned for question i 
worst; is the lowest possible score that can be assigned to 

question i 

35 relweight ; is the relative weight for question i target; is the 
target value for question i 
After the quality scores 192 have been calculated for the 
performance areas 184 in an evaluation 92, the quality 
scores 192 are summed and divided by the sum of the 
40 relative weight for all questions to derive a total evaluation 
quality score 194. Mathematically, this is expressed as: 

n 

qScore-, 

45 ±1 

£ relWeightt 



where 

50 n is the number of applicable questions in the evaluation 
i ranges over the applicable questions in the evaluation 
qScore, is the q-score for question i 
relweight,- is the relative weight for question i 

55 The performance areas 184 may also each include one or 
more expressions 200 to calculate a productivity score 202. 
As previously described, the expression 200 is a user- 
defined formula to calculate a measure of productivity 
against a target score. 

60 The calculation of productivity score 202 for a perfor- 
mance area 184 is a direct computation of the expression 200 
using the productivity data. After the productivity score has 
been calculated for all performance areas 184 that have an 
associated expression 200, the performance area productiv- 

65 ity scores 202 are tallied to derive a total productivity score 
204 for the evaluation. Mathematically, this is expressed as 
a median score of all performance area productivity scores: 
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achieved; 
i target; 



where 

n is the number of performance areas in the evaluation 
having an associated evaluation expression that can be 
successfully calculated 10 
i ranges over the performance areas in the evaluation 
achieved, is the number of points the member actually 
earned for performance area i 
target, is the target value for performance area i 
In another embodiment, the productivity scores for the 15 
different performance areas 184 may be weighted based on 
importance. In this embodiment, the overall productivity 
score for the evaluation is calculated by the following 
equation: 



14 



achievedi 
' targei ; 



20 



25 



30 



where 

n is the number of performance areas in the evaluation 

having an associated evaluation expression that can be 

successfully calculated 
i ranges over the performance areas in the evaluation w,. 

is the importance factor associated with performance 

i's p-score 

achieved is the number of points the member actually 

earned for performance area i 35 
target, is the target value for performance area i 
In operation, the user selects the member 180 and an 
existing guideline or one or more performance areas 184 for 
the evaluation. The evaluation is then automatically gener- 
ated. Quality scores are generated based on the user's 40 
responses; Productivity scores are generated based on pro- 
ductivity data. 

FIG. 9 illustrates details of the filter tables 94. The filter 
allows the user to define certain criteria for reporting and 
charting. This criteria is used to sort the data and display 45 
results in the charts and reports. 

Referring to FIG. 9, the filter tables 94 include a first field 
identifying the filters with a filter ID. The filter ID may be 
any suitable identifier operable to uniquely identify the 
filters. A second field indicates whether the filter includes 50 
plan criteria. Each category of plan criteria is specified by a 
separate look-up table. In a particular embodiment, the 
categories include evaluations, guidelines, performance 
areas, questions, expressions, and evaluators. In this 
embodiment, an evaluation filter table associates the filter 55 
with evaluations, a guideline filter table associates the filler 
with an identified guideline, a performance area filter table 
associates the filter with a specified performance area, a 
question filter table specifies questions for the filter, an 
expressions filter table specifies expressions for the filter and 60 
an evaluators filter table specifies evaluators for the filter. 
Each of these filter's tables include a first field identifying 
the filter with a filter ID and a second field identifying the 
relevant evaluation, guideline, performance area, question, 
expression or evaluator for the filter. 65 

Returning to the filter table 94, a set of date fields specify 
date(s) for the filter. The date may be a current day, current 



month, current quarter, current week, current year, previous 
day, previous month, previous quarter, previous week, pre- 
vious year, or be otherwise defined by the user. Aset of data 
fields specify data for the filter. The data may be members, 
all members, or member levels. A set of averages fields 
specify averages for the filter. The averages may be daily, 
weekly, bi-weekly, or monthly. A set of sample size fields 
specify sample size for the filter. The sample size allows data 
to be filtered based on any sample size entered by the user. 
A keyword field indicates whether the filter includes any 
keywords. Keywords are associated with the filter in a 
separate look-up table as previously described for plan 
criteria. 

Thus, evaluators, guidelines, performance areas, 
questions, expressions, dates, data, averages, keywords, 
sample size, and other criteria may be specified for the filter. 
As a result, a user may narrowly define information and 
obtain meaningful results. 

Although the present invention has been described with 
several embodiments, various changes and modifications 
may be suggested to one skill in the art. It is intended that 
the present invention encompass such changes and modifi- 
cations as fall within the scope of the appended claims. 

What is claimed is: 

1. A method for defining an organizational structure of an 
enterprise in a computerized performance evaluation 
system, comprising: 

storing a plurality of user-defined levels within a com- 
puterized performance evaluation system; 

storing a user-defined hierarchy for the levels within the 
performance evaluation system by storing an assign- 
ment for each lower level in the hierarchy to a higher 
level in the hierarchy; 

storing information identifying a plurality of user-defined 
members within the performance evaluation system; 

assigning each member to a level within the hierarchy to 
define the organizational structure of the enterprise 
within the performance evaluation system; 

storing a plurality of user identifiers within the perfor- 
mance evaluation system, the user identifiers associated 
with particular users that perform evaluations of par- 
ticular ones of the user-defined members; 

generating a first user view for a first user, the first user 
view specifying the particular members to which the 
first user is allowed access; 

generating a second user view for a second user, the 
second user view specifying the particular members to 
which the second user is allowed access; 

assigning first access privileges to the first user, wherein 
the first access privileges allow the first user to perform 
services for user-defined members specified in the first 
user view; and 

assigning second access privileges to the second user, 
wherein the second access privileges allow the second 
user to perform services for user-defined members 
specified in the second user view. 

2. The method of claim 1, wherein the user-defined levels 
are stored by listing the levels in a database table. 

3. The method of claim 1, wherein each lower level is 
assigned to at least a next higher level in the hierarchy. 

4. The method of claim 1, further comprising storing the 
assignments associated with the user-defined hierarchy in a 
database table. 

5. The method of claim 1, wherein the information 
identifying the user-defined members is stored by listing the 
members in a database table. 
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6. The method of claim 1, wherein the members are each 
assigned to a level by storing a user-assigned level for the 
member. 

7. The method of claim 6, wherein the user-assigned 
levels are stored in a database table. 5 

8. The method of claim 6, wherein each of the user- 
defined levels, the user-defined hierarchy, and the user- 
assigned levels are stored in separate database tables. 

9. The method of claim 1, wherein the information 
identifying the user-defined members is stored by listing the 10 
members in a database table and the members are each 
assigned to a level by storing a user-assigned level for the 
member in the database table. 

10. The method of claim 1, wherein the access privileges 
comprise classes of services associated with each user, the 35 
classes of services indicating particular services associated 
with the evaluations that a specified user is allowed to 
perform. 

11. A system for defining an organizational structure of art 
enterprise in a computerized performance evaluation 20 
system, comprising: 

a first database table operable to store a plurality of 
user-defined levels in a computerized performance 
evaluation system; 

a second database table operable to store a user-defined 25 
hierarchy for the levels in the performance evaluation 
system by storing an assignment for each lower level in 
the hierarchy to a higher level in the hierarchy; 

a third database table operable to store information iden- 3Q 
tifying a plurality of user-defined members, each mem- 
ber assigned to a level in the performance evaluation 
system; 

a fourth database table operable to store a plurality of user 
identifiers associated with particular users that perform 35 
evaluations of particular ones of the user-defined mem- 
bers; 

the performance evaluation system operable, for at least 
one of the users, to generate a user view for the user, the 
user view specifying the particular members to which 40 
the user is allowed access; and 

the fourth database table being firer operable to store 
access privileges assigned to the at least one user, 
wherein the access privileges allow the at least one user 



to perform services for user-defined members specified 
in the user view. 

12. The system of claim 11, further comprising the third 
database table operable to store a user- assigned level for 
each of the members. 

13. The system of claim 11, wherein each lower level is 
assigned to a next higher level in the hierarchy. 

14. The system of claim 11, wherein each lower level is 
assigned to each higher level in the hierarchy. 

15. A performance evaluation system, comprising: 
a memory operable to store: 

a plurality of user-defined levels; 

a user-defined hierarchy for the levels by storing an 
assignment for each lower level in the hierarchy to a 
higher level in the hierarchy; 

information identifying a plurality of user-defined 
members, each member assigned to a level; 

a plurality of user identifiers associated with particular 
users that perform evaluations of particular ones of 
the user defined members; and 
a processor coupled to the memory and operable to: 

generate a first user view for a first user, the first user 
view specifying the particular members to which the 
first user is allowed access; 

generate a second user view for a second user, the 
second user view specifying the particular members 
to which the second user is allowed access; 

assign first access privileges to the first user, wherein 
the first access privileges allow the first user to 
perform services for user-defined members specified 
in the first user view; and 

assign second access privileges to the second user, 
wherein the second access privileges allow the sec- 
ond user to perform services for user-defined mem- 
bers specified in the second user view. 

16. The performance evaluation system of claim 15, 
wherein the memory is further operable to store a user- 
assigned level for each of the members. 

17. The performance evaluation system of claim 15, 
wherein each lower level is assigned to a next higher level 
in the hierarchy. 



07/21/2004, EAST Version: 1.4.1 



